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Abstract 

Filtering  and  smoothing  within  a  convex  and  closed  family  of  stationary  processes 
is  considered.  The  perf romance  criterion  adopted  is  the  mean  square  error.  Using 
a  saddle  point  game  approach,  we  develop  two  classes  of  lower  bounds  for  this  error. 
Those  bounds  can  be  used  as  evaluation  measures  in  the  design  of  robust  filters  and 
smoothers . 
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1. 


Introduction 


The  filtering  and  smoothing  problems  have  enjoyed  attention  for  a  large  number 
of  years.  The  bulk  of  the  work  concentrates  around  the  parametric  model  [2],  The 
assumption  there  is  that  the  measures  which  generate  the  data  and  noise  processes 
are  well  known.  Also,  the  term  filtering  usually  refers  to  the  extraction  of 
information  data  from  noisy  observations,  while  the  term  smoothing  usually  refers 
to  the  removal  of  outliers  strictly  from  information  data.  Unfortunately,  the  two 
terms  have  been  used  sometimes  inter-changeably  in  the  literature.  Finally,  within 
the  framework  of  the  parametric  model,  the  bulk  of  the  work  has  focused  on  linear 
filters  and  smoothers,  due  to  the  mathematical  tractability  of  the  problem  then. 

The  last  seven  years  considerable  attention  has  been  given  to  the  robust 
filtering  and  smoothing  problems.  The  assumption  there  is  that  either  the  process 
that  generates  the  information  data,  or  the  noise  process,  or  both  are  statistically 
contaminated.  That  is,  the  stochastic  measures  that  generate  the  processes  are 
only  members  of  a  whole  class  of  measures,  rather  than  being  well  known.  Tukey  [3] 
introduced  a  variety  of  empirical  robust  smoothers,  among  which  the  "L-smoothers" 
(smoothers  based  on  moving  order  statistics)  prevail.  Velleman  [9]  studied 
numerically  the  performance  of  the  L-smoothers,  for  information  processes  consisting 
of  a  pure  sinusoidal  and  additive  noise  with  various  outliers.  Another  class  of 
robust  smoothers  is  based  on  robustified  splines,  where  the  basic  reference  on  this 
approach  is  provided  by  Huber  [1].  The  smoothers  in  [1],  [3],  and  [9]  are  mostly 
valuable  in  exploratory  data  analysis,  where  conclusions  on  nonobvious  structural 
properties  of  the  information  process  are  sought.  On  the  other  hand,  Martin  con¬ 
sidered  smoothers  whose  objective  is  either  the  generation  of  outlier-free  time 
series  for  fitting  autoregressive-moving-average  models  [5],  [6],  or  the  estimation 
of  spectral  density  functions  [14],  Then,  in  contrast  to  the  methods  in  [1],  [3] 
and  [9],  more  detailed  knowledge  on  the  statistical  behavior  of  the  smoothers  is 


needed.  The  smoothers  proposed  in  [1],  [3],  [9].  [5],  [6],  and  [14]  are  in  general 
nonlinear.  A  nice  review  on  these  smoothers,  as  well  as  an  additional  class  of  such 
smoothers  and  interpolators  is  provided  by  Martin  [7].  Using  the  terms  smoothing 
and  filtering  in  their  usual  context  (as  explained  in  the  beginning  of  the  introduc¬ 
tion),  we  can  refer  to  studies  on  the  asymptotic  robust  behavior  of  some  linear 
smoothers  and  filters.  Stuck  [8]  studied  the  minimum  error  dispersion  in  linear 
filtering,  within  the  class  of  symmetric  stable  processes.  Hosoya  [13]  considered 
predictive  linear  smoothing  within  the  convex  class  of  processes  that  is  modeled 
by  linear  contamination  of  a  well  known  nominal  process.  He  formalized  the  problem 
as  a  saddle  point  game  in  the  frequency  domain,  and  found  the  saddle  point  solution 
in  terms  of  spectral  densities.  For  some  work  on  robustif ication  of  Wiener  filtering 
see  [16]. 

Despite  the  work  on  robust  smoothing  and  filtering  mentioned  above,  a  general 
theory  concerning  the  problem  is  still  lacking.  This  lack  makes  the  comparison  of 
different  smoothing  and  filtering  schemes  virtually  impossible.  A  first  effort 
towards  that  direction,  for  nonlinear  smoothers,  is  provided  by  Mallows  [15],  and 
it  is  further  carried  on  by  Martin  [7].  Mallows'  work,  however,  is  directed  towards 
the  analysis  of  robust  smoothers  rather  than  addressing  the  issue  of  how  to  design 
such  smoothers.  The  design  issue  for  both  smoothing  and  filtering  is  perhaps  better 
addressed  if  the  qualitative  aspects  of  robustness  are  first  carefully  considered. 
Indeed,  through  the  appropriate  selection  of  performance  measures,  the  theory  of 
qualitative  robustness  provides  sufficient  conditions  that  can  be  used  as  design 
guidelines.  For  the  class  of  memoryless  processes,  these  sufficient  conditions 
were  provided  by  Hampel  and  they  can  be  found  in  [1],  For  the  class  of  stationary 
processes  with  memory,  the  formalization  of  qualitative  robustness  and  the  sub¬ 
sequent  sufficient  conditions  can  be  found  in  [10],  A  first  qualitative  Normaliza¬ 
tion  of  the  filtering  problem  is  presented  in  [12].  There,  the  approach  used  in 
[10]  in  conjuction  with  results  from  [11]  are  used  and  further  extended,  to  provide 


sufficient  conditions  for  robust  filtering.  As  in  robust  parameter  estimation,  the 
results  in  [12]  declare  that  linear  filtering  or  smoothing  may  be  highly  nonrobust. 

On  the  other  hand,  an  appropriate  selection  of  a  memoryless  nonlinearity  followed 
by  a  linear  filter  may  be  robust.  (Such  a  selection  is  presented  in  [12]  for 
filtering  in  additive,  Gaussian,  and  memory less  noise).  Therefore,  Hosoya's 
approach  [13]  may  be  nonrobust  for  some  class  of  processes,  if  some  nonlinear  trans¬ 
formation  of  these  processes  does  not  preceed  the  linear  filtering.  That  is,  for 
small  deviations  within  the  class  of  the  processes,  (the  measure  rather  than  the 
spectrum)  unreasonably  large  performance  deviations  may  appear.  In  addition,  as 
compared  to  the  performance  of  a  linear  filter  or  smoother,  the  performance  induced 
by  an  appropriate  nonlinear  such  selection  may  be  superior  for  all  processes  in  the 
considered  class.  Indeed,  given  a  well  known  process,  the  matched  linear  filter 
or  smoother  is,  in  general,  the  worst  in  terms  of  performance. 

In  the  present  document,  we  consider  the  filtering  problem  in  a  generalized 
fashion.  Specifically,  we  consider  the  problem  of  extracting  the  data  generated  by 
a  stationary  information  process  with  memory,  from  noisy  observations,  in  general. 

We  include  the  case  of  no  noise  in  our  general  formalization;  thus  we  include  the 
smoothing  problem.  We  assume  that  the  information  and  noise  processes  (the  latter 
may  be  absent)  are  jointly  stationary,  and  we  adopt  the  mean  square  performance 
criterion.  We  assume  that  the  noise  process  (if  present)  is  well  known,  and  we 
allow  the  information  process  to  vary  within  a  prespecified  convex  and  closed  family 
of  stationary  processes.  We  consider  the  class  of  filters  or  smoothers  that  consist 
of  a  stationary  (in  general  nonlinear)  operation  on  observation  data  within  a  sliding 
window  of  fixed  and  finite  length,  followed  by  a  linear  transformation.  We  first 
show  that  for  every  fixed  nonlinear  operation,  there  exists  a  saddle  game  formali¬ 
zation  and  a  unique  saddle  point  solution.  Then,  we  develop  two  classes  of  lower 
bounds  on  the  saddle  point  error  induced  by  every  choice  of  the  nonlinear  operation. 
We  use  the  saddle  point  error  induced  by  the  class  of  linear  filters  or  smoothers 
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as  an  upper  bound  to  the  saddle  point  error  induced  by  all  possible  nonlinear 
operations.  The  saddle  point  error  performance  of  any  given  nonlinear  operation 
is  then  evaluated  against  the  developed  lower  bounds  and  the  above  upper  bound. 

2 .  The  Model-Notation 

Let  [y,R,X]  denote  a  stationary  process,  with  measure  y,  name  X,  and  R  the 
real  line  representing,  in  general,  the  alphabet  of  the  process.  Let  [R,V,R] 
represent  a  stationary  channel  with  input  and  output  alphabets  represented,  in 
general,  by  the  real  line.  Given  y  and  V  as  above,  we  denote  by  [yV  \r,Z]  the 
stationary  process  induced  by  the  stationary  process  [y,R,X]  and  the  stationary 
channel  [R,v,Rj.  In  the  filtering  and  smoothing  problems,  [y»R,X]  represents  the 
information  process  whose  data  sequences  must  be  extracted.  The  channel  [R,V,R] 
represents  the  transmission  or  observation  noise  and  in  the  smoothing  problem  it 
is  deterministic.  The  process  [yv  \r,Z]  represents  the  observation  process  in 
both  the  filtering  and  smoothing  problems.  Any  filter  or  smoother  operates  then  on 
data  sequences  generated  by  the  process  [yV  \r,Z].  We  will  denote  by  X^,  Z the 
random  variables  representing  the  ith  datum  from  the  information  and  observation 
processes  respectively.  We  will  denote  by  x^,  z ^  specific  values  of  the  random 
variables  X^,  respectively.  We  will  also  denote: 


xji  =  {V  xw . V  *  J*1 

Zi  =  ^Zi’  Zi+l”’*’Zj}  ’  j  -  1 

X1  =  {V  xui’*-*’xj}  ’  3  *  i 

zi  =  {zi’  zi+l”-”zj}  ;  J  t  i 


(1) 


We  assume  that  the  measure  y  and  the  measures  induced  by  the  stationary  channel 

[R,V,R]  are  absolutely  continuous.  Then,  we  denote  by  f  (z^  *)  the  (j -i)-dimensional 

y ,  v 

density  function  induced  by  the  process  [yv  \r,Z]  at  the  point  z|  in  the  (j-i)- 


dimensional  Euclidean  space.  We  will  use  the  symbol  E  for  expected  value.  Then, 

will 

denote  the  conditional  expectation  of  the  random  variable  X^  conditioned  on  the 
observation  sequence  Z^,  as  induced  by  the  process  [y,R,X]  and  the  channel  [R,V,R]. 
For  convenience  in  notation,  we  will  denote: 


Xq  will  denote  the  expected  value  of  Xq  at  the  measure  y,  and  E^  ^ 


-il 


Vv.Ci.j)1 


=  E, 


z-:  =  zi 


(2) 


Let  us  consider  a  sliding  block  window  of  length  t,  operating  on  observation 
sequences.  Let  k  denote  the  number  of  sliding  steps  per  shift  of  the  sliding  block 
window,  where  1  <  k  <^  •£.  If  k  =  1,  there  is  an  one  datum  sliding  per  unique  shift 
of  the  sliding  window.  If  k  =  £,  each  unique  shift  of  the  sliding  window  corresponds 
to  a  shift  by  a  whole  £-size  data  block.  This  last  case  corresponds  to  a  block, 
rather  than  sliding  block  operation.  For  given  k  and  t,  we  will  denote: 


B,(k)  (/)  =  E  i  X0 

y,v  j 


,ik  , 

'ik+l-£ 


=  z 


(3) 


;  where  z  denotes  a  given  value  of  an  -C-dimensional  data  vector. 
We  will  also  denote  then: 


"I.v.W.i.J6* 


=  [Vk) 

L  i.y.v 


,  Is 
(z  ),., 


,&fk)  (zl) 

*  j.y.v  . 


(4) 


;  where  the  expression  in  [A]  symbolizes  a  row  vector,  and  T  means  transpose. 

As  we  already  mentioned,  the  stationary  channel  [R,V,R]  may  be  deterministic. 


Then,  there  is  no  observation  noise  and  the  data  extraction  problem  becomes  the 
smoothing  problem.  Thus,  in  our  general  formalization,  with  the  inclusion  of  the 
channel  [R,v,R],  both  the  filtering  and  smoothing  problems  are  included.  Also,  our 
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model  includes  the  filtering  problem  for  a  process  embedded  in  additive  noise, 
with  the  noise  and  information  processes  being  mutually  independent. 

In  general,  our  objective  is  the  design  of  an  appropriate  smoother  or  filter. 

We  will  consider  the  existence  of  an,  in  general,  sliding  block  window  on  the 

observation  sequences...,  Z^,  Z^,  Z,...  .  We  will  denote  by  Z  the  size  of  the 

window,  and  by  k  the  number  of  data  steps  per  single  window  sliding;  where  1  <  k  <  Z. 

We  will  denote  by  g^  some  stationary  operation  on  £-size  data  blocks.  Then,  we  will 
consider  filters  and  smoothers  of  the  following  form: 


<2>  'I)  ai  *£  '  1  ±  k  1  * 


;  where  z  denotes  an  infinite  sequence  ...,z  ^,Zq,z^,...,  and  {a^  is  a  set  of 
constant  coefficients. 

The  filter  and  smoother  form  in  (5)  clearly  represents  a  cascade  operation  of 
an,  in  general,  nonlinear  operation  (represented  by  g^)  and  a  linear  filter  or 
smoother  (represented  by  {a^}).  If  Z  -*■  <»  and  is  allowed  to  move  within  the 
total  class  of  all  possible  operations,  the  cascade  operation  in  (5)  clearly 
represents  the  totality  of  possible  operations  on  the  observation  data.  If  Z  -  1, 
and  any  operation  per  single  datum,  the  operation  in  (5)  includes  linear  filters 
and  smoothers.  Therefore,  the  cascade  operation  in  (5)  represents  (for  varying 
h,£,g^,  (a^})  all  possible  filters  and  smoothers,  with  both  sliding-block  and 

block  operational  characteristics. 

As  mentioned  before,  the  objective  of  the  smoother  or  the  filter  is  the 

extraction  of  the  data  from  the  information  process  [p,R,X],  Due  to  the  stationarity 

of  the  model,  we  will  assume  without  lack  in  generality  that  the  operation 

G  .  /  i(z)  in  (5)  corresponds  to  a  mapping  of  the  datum  Xn  from  the  process 
g £ ,  k  » t  a  ^  /  U 

(p,R,X].  Then,  adopting  the  mean  square  performance  criterion,  we  will  use  the 


following  notation  for  the  resulting  mean  square  error: 


e(y»V,  k,g<t,{ai})  E^v  jxQ  "  ^  g^  (Z^+1_^)j  (6) 

j 

For  Z  -*■  00  the  parameters  k  and  {a^}  become  absolute.  Then,  we  will  denote  the 
error  in  (6)  as  e(y,V,  gM). 

We  will  denote  by  F(k,g^)  the  class  of  filters  or  smoothers  described  by  (5) 
for  given  g^,  Z,  and  k.  This  class  is  then  generated  by  variations  of  the  set 
{a^}.  We  will  denote  by  F(k,£)  the  class  that  is  described  by  (5)  for  given  k  and 
Z.  The  class  F(k,£)  is  generated  by  variations  of  both  g^  and  {a^}.  We  will  denote 
by  F  the  class  described  by  (5)  for  Z  00 .  This  last  class  clearly  includes  all 
possible  filters  and  smoothers,  and  also: 

F(k,g«)  C.  F (k,£) C_  F  ;  V  Z 

(7) 

F(k ,Z)  ~2_  F(k,£.m)  ;  V  Z  ;  V  m  >  ;  ¥  k 

Let  F(L)  denote  the  class  FCl.g^)  for  g(x)  =  x.  Then,  due  to  the  relation¬ 
ships  in  (7),  we  have: 

F(L)C  F(l,l)3  F(k,£)  ;  Vk:l  <  k  <  Z\  V  Z  >  1  (8) 

Therefore,  for  the  error  expression  in  (6)  we  have: 


inf 


eCy.v.k.g^,^})  >  inf  e  (y  ,V,1  ,g^  ,  {aj  )  > 


V-V  e  F(L) 


>  inf  e(y,v,k,g£, {a^} ) 

G  .  r  -i  G  F 
g£,k,iai} 


=  inf  e(y,v,  g^)  ;  V  Z 
G  e  F  ;  V  k:  1  <  k  <  I 

8oo 

(9) 


Thus,  we  can  express  the  following  proposition: 


For  every  given  y  and  V,  the  class  F(L)  of  linear  filters  and  smoothers 
represents  a  worst  case.  Specifically,  the  infimum  of  the  mean  square  error  in 
(6)  within  this  class  is,  in  general,  larger  than  the  parallel  infimum  within 


the  class  F(l,l).  The  global  infimum  is  achieved  within  the  class  F. 

Proposition  1  expresses  formally  what  we  stated  informally  in  the  introduct: 
That  the  class  of  linear  filters  and  smoothers  can  be  used  to  represent  an  upper 
bound  on  the  mean  square  error. 

In  the  present  document,  we  are  interested  in  robust  smoothing  and  filtering. 
We  thus  consider  a  family  M  of  measures  y,  rather  than  a  well-known  measure  y^. 

For  convenience  in  notation,  we  will  assume  that  the  noise  channel  [R,V,R]  is  well 
known.  In  our  general  derivations,  however,  this  assumption  is  not  necessary.  The 
only  assumption  we  need  there  is  just  the  consideration  of  a  convex  and  closed 
family  of  joint  measures. 

Given  a  well-known  noise  measure  V  (that  may  be  deterministic)  and  a  convex 
and  closed  family  M  of  information  processes  y,  we  will  model  robustness  as  a 
saddle  point  game,  and  we  will  search  for  saddle  point  solutions.  We  will 
initially  ignore  the  qualitative  aspects  of  robustness,  since  we  are  basically 
searching  for  performance  lower  bounds  within  the  classes  M  and  F.  We  will  come 
back  to  qualitative  robustness  later. 

We  conclude  this  section  by  noticing  that  all  the  sets  F(L) ,  F(k,g^),  F(k,£) , 
and  F  in  (7)  are  convex,  and  by  stating  the  general  formalization  for  a  saddle 
point  game.  Given  two  sets  M  and  S,  given  a  payoff  function  K(y,A);  where  yeM  and 
AeS,  the  game  on  MxS  with  payoff  K(y,A)  has  a  saddle  point  solution  if  and  only  if 
there  exists  a  pair  (y*,A*)  e  MxS  such  that: 
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VyeM  ;  K(y,X*)  <  K(y*,X*)  <  K(y*,X)  ;  V  XeS 


(10) 


The  saddle  point  solution  is  unique  if  and  only  if  the  pair  (y  ,X  )  that  satisfies 


&  ic 

(10)  is  unique.  The  quantity  K(y  ,X  )  is  called  the  saddle  value  of  the  game. 


3 .  The  Game  for  Fixed  Nonlinearity 

Let  be  a  fixed,  well-known  stationary  and  deterministic  operation.  Let  t 
be  known  and  finite.  Then,  let  us  consider  the  convex  class  F(k,g^)  of  filters  and 
smoothers,  where  F(k,g^)  is  defined  in  the  previous  section  and  k  is  assumed  fixed 
and  known.  Let  the  noise  channel  [R,V,R]  be  well  known,  and  let  M  be  a  convex  and 
closed  family  of  information  processes  y.  We  will  search  for  a  saddle  point  game 
formalization  and  solution  on  MxFCk.g^),  using  as  the  payoff  function  the  error 
e(y,V,k,g£, {a_^})  in  (6).  We  will  first  consider  finite  sequences  of  observation 
data,  and  then  asymptotically  long  such  sequences. 

Ic 

Let  the  smoother  or  filter  operate  on  the  limited  length  sequency  z_nk+q_£ 
of  observation  data.  Then,  the  smoother  or  filter  take  the  form: 


m-n-1 


,  (m-n-1) k. 
g^.k-.ta^}  nk+l-C 


r 

a.  H  (a. 


ik+l-l 


)  ;  m  >  1 


(11) 


i=-n 


Define : 


(a)  ■  [g4(^+1^> 


n  <  i  <  m-n-1] 


(12) 


R(y,v,g^) 


(13) 


;  where  the  expression  in  (12)  denotes  a  row  vector.  Then,  it  is  straightforward 
to  obtain  the  following  result: 
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inf  e(y  ,v,k,g£,  {a^» 

Vk-'ai> e  F(k'8^ 


EU,V  4 


em,v  |xo  vV2>!  ,rVv>g*>  ep,v  |xo  vH 


;  where  Z  denotes  the  sequence  . 

The  infimum  in  (14)  is  actually  realized  for: 


u?  ■  < 


W,g<  ■  E'1  Em,v  jX0  V(2>| 


;  where  A  denotes  the  column  vector  of  the  appropriate  coefficients 

y,v,g£ 

{a.  ;  -  n  <  i  <  m-n-l}. 
x  —  — 

Let  us  consider  the  game  on  MxFCk.g^)  with  the  payoff  function  e(y,V,k,g£, 
{a^  ;  -  n  <  i  <  m-n-l}) .  We  will  show  that  the  game  has  a  saddle  point  solution. 
We  will  do  that  in  two  steps . 

Let  us  denote: 

e(y,v,k,g^)  =  inf  e(y ,v,k,g^,{ai»  (16) 

Ggrk,<a.}  E 

As  the  first  step  in  the  proof  for  the  solution  of  the  game,  we  present  the 
following  lemma: 

Lemma  1 

The  error  e(y,v,k,g£)  in  (16)  is  concave  in  M.  Therefore,  there  exists  some 
y*  in  M  such  that: 


e(y  ,V,k,g„)  =  sup  e(y,v,k,g£) 
yeM 


The  proof  of  lemma  1  is  in  appendix  A. 

The  measure  y  that  satisfies  the  supremura  in  lemma  1  may  not  be  unique.  It  is 
unique,  however,  within  a  class  M  of  measures  that  induces  distinct  expectations 
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E  \xn  V  (Z) |  and  matrices  R(y,V,g»). 
h.V  (  0  g£  )  *■ 

We  express  now  the  second  step  in  the  proof  for  the  existence  of  a  solution 
to  the  game,  in  the  following  theorem. 


Theorem  1 

Let  us  consider  the  game  on  MxF(k,g£>  with  the  payoff  function  e(y,v,k,  g^, 
{a.  ;  -  n  <  i  m-n-l})  .  The  game  has  a  saddle  point  solution.  That  is,  there 
exists  a  pair  (y  ,{ai))  in  MxF(k,g^)  such  that: 

VyeM  ;  e(y,v,k,g^,{a*})  <  e(y*,v,k,g^,{a*»  <  e(y*,v,k,g^,{ajL}) 


The  pair  (y*,{a*}) 


is  such  that: 


;  V  {a^}  e  F(k,g^) 


;  where  A 


e(y*,v,k,g^,{a*})  =  e(y  ,v,k,g^)  =  sup  e(y,v,k,g^) 
{ai}  “  A\ 

y 

is  given  by  (15) . 


u*’v,8£ 


According  to  the  theorem,  to  find  a  folution  of  the  game,  it  is  sufficient 
to  first  express  the  infimum  in  (16)  for  every  y  in  M,  and  to  then  find  the  supremum 
of  the  resulting  expression,  with  respect  to  y  in  M.  The  proof  of  the  theorem  is 
in  appendix  A,  and  a  unique  solution  exists  if  uniqueness  in  lemma  1  is  satisfied. 

We  will  now  consider  the  asymptotic  case,  where  the  observation  sequences  used 
in  the  smoothing  or  filtering  extend  to  the  infinite  past  and  the  infinite  future. 
That  is,  we  will  in  general  consider  noncausal  filters  and  smoothers.  This  is  done 
for  convenience  here.  Some  causal  filters  of  predictive  nature  and  belonging  to 
the  general  class  F(l,g^)  have  been  considered  in  [12]. 


Let  the  filter  or  smoother  be  allowed  the  following  asymptotic  form: 


G  .  r  i (Z) 

grk’{ai} 


'  Z  ai  ^(z^+l-£> 


Then,  define: 


h(w)  =  ak  e’jkw 


s  (w)  =  y 

y,v,g£. 

m=-°° 


e-JmW  E 


j  e  (7°  )  a*  Czmk  )! 

y,v  |  gV  1-V  gl  mk+l-T  j 


;  where  E^v  J  H  (Zmk+l-£) j  =  f(i_m)’  due  to  the  st^±onarity  of  y,V. 

and  the  operation  g^,  and  where  *  means  conjugate. 


Also,  define: 


L(k) 


sp,v,8£.b  <»>  -  E  »'J"”  ep,v  ,  C.v <z  >  4 


fy 

;where  Z  denotes  some  ^-dimensional  vector  of  consequtive  data  from  the  observation 
(k)  l 

process,  and  B  (z  )  the  conditional  expectation  defined  in  (3) . 

V  9  V 

The  expressions  in  (18),  (19),  and  (20)  clearly  signify  Fourier  transforms, 
where  H(w)  is  the  asymptotic  representation  of  the  set  {a^}  of  coefficients.  Sub¬ 
stituting  the  error  expression  e(y ,v,k, g^, {a^})  in  (6)  by  e(y ,v,k,g^,H)  in  this 

«n* 

case,  we  easily  find  that  the  Fourier  transform  H  (w)  corresponding  to  the 

A 

vector  A  in  (15)  is  given  by  the  following  expression. 

,8 1 

S  q  (w) 

*  y.v.g^B 

H*  „  (w)  =  -x - Vy-  (21) 

SM,v.8,(w) 


13 


Also,  the  infimum  e(y,V,k,g£)  in  (16)  takes  now  the  following  form. 


e(y,V,k,g£> 


(22) 


Since  the  length  of  the  observation  sequences  used  by  the  filter  or  smoother 
was  never  used  in  the  proof  of  theorem  1,  we  can  express  a  theorem  parallel  to  it 
for  the  asymptotic  case. 

Theorem  2 

with  the  payoff  function  e(y ,v,k,g£,H)  has  a  saddle 

&  -k 

point  solution.  That  is,  there  exists  a  pair  (y  ,H  )  in  MxF(k,g£)  such  that: 

VyeM  ;  e(y,v,k,g£,H*)  <  e(y*,v,k,g£,H*)  <  e(y*,v,k,g^,H) ;  V  H  e  F(k,g£) 

The  pair  (y*,H*)  is  such  that 

e(y*,V,k,g£)  =  sup  e(y,v,k,g„) 
yeW 

H*(w)  =  H**  (w) 

y  ,v,g^ 

and  H*  ,  (w)  are  given  by  (22)  and  (21)  respectively, 

y.v.g^ 


;  where  e(y,V,k,g^) 


The  game  on  MxF(k,g£) 


In  this  section,  we  showed  that  if  in  the  cascade-type  filter  or  smoother  in 
(5)  we  fix  the  operation  g^,  we  can  formulate  a  saddle  point  game  in  MxF(k,g£) 
whose  solution  exists.  Furthermore,  this  solution  corresponds  to  mean-square 
error  minimization  in  the  worst  case.  It  does  not  necessarily  guarantee  perfor¬ 
mance  continuity  within  the  class  M.  For  such  continuity  the  sufficient  conditions 
of  the  qualitative  robustness  [10,11,12]  should  be  satisfied.  That  can  be  accomplished 
through  the  appropriate  selection  of  the  operation  g^,  and  it  may  also  require  that 


only  a  subset  of  the  set  F(k,g^)  be  considered  in  the  filter  or  smoother  selection. 
An  example  of  the  above  can  be  found  in  [12]. 


4 .  The  Unrestricted  Game 

In  this  section,  we  will  consider  the  case  where  the  filter  or  smoother  is 
described  by  (5)  for  Z  00 .  That  is,  the  class  F  of  section  2  is  adopted.  We  will 
search  for  a  game  formalization  and  solution  on  MxF;  where  M  a  convex  and  closed 
family  of  measures  y.  As  in  section  2,  we  will  use  the  symbol  to  signify  filters 
and  smoothers  in  F.  We  will  use  the  symbol  even  in  the  case  where  the  filter  or 
smoother  operates  on  observation  sequences  of  finite  length. 

Let  the  filter  or  smoother  operate  on  the  observation  sequence  Z^.  The,  we 
will  denote  as  in  (2): 


a 


y,v,(i,j) 


uJ-b  & 


lx. 


zi 


= 

i 


(23) 


Given  some  measure  y  in  M  it  is  then  well  known  that: 


inf  .Oi.v.o  -  eCM,V,0)j>v>(iiJ))  -  Eu  X-  -  EUjV  (24) 


g  eF 


;  where  e(y,v,gOT)  the  mean  square  error  in  (6)  for  Z  ->  °°. 

Using  the  payoff  function  e^V.g^),  we  are  searching  for  a  saddle  point  solution 
on  MxF.  As  in  section  3,  we  proceed  in  two  steps.  The  first  step  is  represented 
by  the  following  lemma. 

Lemma  2 

The  error  e(y,v,a  .  ..)  in  (24)  is  concave  in  M.  It  is  strictly  concave 

iff  for  any  two  y1 ,y„  in  M  the  conditional  expectations  a  . .  .  (z^  *)  and 

i “i  ^  •  • 

Ct, ,  ^  ^(zJ  )  (23)  are  not  equal  almost  everywhere  on  1 
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i 

3 

► 

3 

! 


The  proof  of  lemma  2  is  in  appendix  B. 

We  now  show  the  existence  of  a  saddle  point  solution  of  the  game  on  MxF  with 
payoff  function  eCy.V.g^),  with  a  theorem. 

Theorem  3 

Consider  the  game  on  MxF  with  payoff  function  efy^g^)  .  The  game  has  a 
saddle  point  solution.  That  is,  there  exists  a  pair  (y*,g*)  on  MxF  such  that: 


VyeM  ;  e(y,v,g^)  <  e(y*,v,g*)  <  e(y*,vfgco)  ;  V  gro  e  F 

;  where  eCy.v.g^)  is  given  by  (6)  with  t  00 . 

The  pair  (y  ,  gOT)  is  such  that: 

U*  :  -  sup  .(U.v.a,^  (1  fJ)) 

£  - a  * 


j i — i 

If  there  are  no  yi»y2  ln  M  such  that  ay  v  (i  j)^Z  ^  and  °y  V  (i  j)^Z  ^ 

i-i  1  *  '*  2 

are  equal  almost  everywhere  on  R  ,  the  pair  (y  .g^)  is  also  unique. 


The  proof  of  the  theorem  is  in  appendix  B.  Since  in  the  proof,  the  dimen¬ 
sionality  (j-i)  is  no  where  used,  the  statement  holds  also  asymptotically. 
According  to  the  theorem,  to  find  the  solution  of  the  game,  it  is  sufficient  to 
find  the  minimum  mean  square  error  in  F  for  each  measure  y,  and  to  then  find  the 
supremum  of  the  result  on  M. 

It  is  clear  that  since  F(k,g^)C~F  ;  Vk,Z,g^,  the  error  e(y,v,k,g£)  in  (16) 

is,  in  general,  bounded  from  below  by  the  error  e(y,V,a  .,)  in  (24)  for 

y,V,(i,j) 

fixed  Z^,  and  for  every  k,Z,g^,  and  y  in  M.  Therefore,  the  supremum  of  e(y,V,k,g£) 
on  M  is  also  then  bounded  fiom  below  by  the  supremum  of  e(y,V,a  , ,  .-.)  on  M. 
Thus,  we  can  express  the  following  corollary. 


ib 


Corollary  1 

Given  a  convex  and  closed  class  M  of  measures  y,  given  the  sequence  of 

observations  on  which  the  filter  or  smoother  operates,  the  saddle  value 

e(y*,v,a  A  )  of  the  unrestricted  game  in  theorem  3  bounds  from  below  the 

V  ,V,(i,j) 

saddle  value  e(y*,V,k,g£)  of  the  restricted  game  in  theorem  1,  for  every  k,£,  and 

* 

g»  choice.  Thus,  e(y  ,V,a  *  )  acts  as  a  global  lower  bound  on  the  saddle 

y  ,v,(i,j) 

value  performance  of  all  the  filtering  and  smoothing  schemes. 


We  will  conclude  this  section  with  the  presentation  of  the  solution  of  the  game 
in  theorem  3,  for  two  important  classes  M  of  information  processes.  We  present  the 
results  in  two  lemmas,  and  we  include  the  proof  of  the  first  lemma  in  appendix  B. 

Lemma  3 

Let  the  noise  channel  [R,V,R]  be  either  deterministic,  or  Gaussian,  stationary, 
and  additive.  If  the  latter  is  true,  let  also  the  process  v  be  independent  of  any 
information  process.  Let  M  be  the  convex  and  closed  family  of  stationary  processes 
with  equal  means  and  spectral  densities.  Then,  the  solution  of  the  game  in  theorem 

•Jlp 

3  is  provided  by  the  Gaussian  process  y  and  the  corresponding  linear  filter  or 

smoother  a  .  This  is  true  for  any  dimensionality  (j-i)  of  the  filter  or 

V  ,v 

smoother. 


We  observe  that  the  saddle  point  solution  of  the  game  in  theorem  3,  within 
the  class  of  measures  with  equal  spectral  densities,  is  satisfied  by  the  maximum 
entropy  measure  within  the  class.  The  relationship  between  worst  error  and  highest 
entropy  seems  natural.  In  qualitative  terms,  a  high  entropy  process  is  less  pre¬ 
dictable.  Thus,  any  effort  for  the  extraction  of  its  data  should  result  in  higher 


error . 
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In  the  proof  of  lemma  3,  in  appendix  B,  we  denoted  by  e  (y,v)  the  infimum  of 

la 

the  error  in  (6),  on  the  class  F(L)  of  linear  smoothers  and  filters,  and  at  the 
measure  y.  The  quantity  e„ (y,v)  is  a  function  of  the  y  and  V  spectral  densities 

Jhi 

only,  if  both  the  measures  y  and  V  have  zero  mean  and  are  mutually  independent. 

Let  us  consider  a  convex  and  closed  family  S  of  spectral  densities,  and  let  us 
assume  that  e^(y,v)  is  strictly  concave  with  respect  to  the  spectral  density  of 
the  measure  y.  This  is  in  general  true  for  the  channel  model  in  lemma  3  (see  ref. 
[13]).  Let  us  consider  the  convex  and  closed  family  M  of  measures  y,  such  that 
each  y  is  zero  mean,  stationary,  and  has  spectral  density  in  S.  Let  us  denote  by 
f  the  spectral  density  of  the  measure  y.  Then,  there  exists  a  unique  spectral 
density  f  in  S,  such  that: 


eL(f*‘fV)  =  fP  eL(V  V  (25) 

;  where  fy  is  the  spectral  density  of  the  measure  v,  and  we  have  represented 
eL(u,v)  by  e^ ( f  ,  f^)  instead. 

It  is  now  clear  from  the  approach  used  in  the  proof  of  lemma  3,  that  we  can 
express  the  following  lemma. 


Lemma  4 

Let  the  channel  [R,V,R]  and  its  relationship  to  any  information  process  be 

as  in  lemma  3.  Let  S  be  a  convex  and  closed  family  of  spectral  densities,  and 

let  the  error  e  (f  ,  f  )  be  strictly  concave  on  S  sith  respect  to  f  .  Let  M  be 
L  y  v  y 

the  convex  and  closed  family  of  stationary,  zero  mean  processes,  with  spectral 
densities  in  S.  Let  the  measure  \>  be  zero  mean.  Then,  the  solution  of  the  game 
in  theorem  3  is  provided  b;T  this  Gaussian  measure  y  whose  spectral  density  f  A 

y 


e  (f  ft,f  )  =  sup  e  (f 
y  V  f MeS 


) 


satisfies: 


L8 


V- , 


From  lemma  4,  it  is  clear  that  if  the  unrestricted  family  F  of  smoothers  and 
filters  is  considered,  the  solution  of  a  saddle  point  game  within  a  convex  and 
closed  family  of  spectral  densities  is  only  an  intermediate  step  towards  the 
solution  of  the  larger  game.  Due  to  the  relationships  in  (8),  tho  above  statement 
holds  even  if  F  is  substituted  by  F(k,£). 

5.  The  Game  in  F(k,£) 

To  this  point,  we  have  analyzed  two  extreme  cases.  In  sectim  3,  we  fixed 
the  parameters  k  and  Z  as  well  as  the  nonlinearity  g^  and  we  analyzed  the  game  in 
the  class  F(k,g^)  of  smoothers  and  filters  in  (5).  In  section  4,  we  analyzed  the 

unrestricted  game  in  the  global  class  F  of  filters  and  smoothers.  Here,  we  will 

consider  an  intermediate  case.  Specifically,  we  will  assume  that  the  length  Z  and 
the  sliding  step  k  of  the  sliding  block  window  in  (5)  are  fixed,  and  that  the  non 
linearity  g^  and  the  set  {a^}  of  coefficients  may  vary.  Thus,  we  consider  the 
class  F(k ,Z)  of  smoothers  and  filters.  In  this  case,  we  can  not  always  formulate 
a  game  in  the  strict  sense.  Our  approach  is  somewhat  different  than  the  previous 
approaches,  and  we  first  present  the  gist  of  it. 

Let  us  denote  by  the  class  of  all  possible  operations  g^.  For  some  y  and 

g^,  consider  the  infimum  e(y,v,k,g£)  in  (16).  For  every  y,  we  will  find  some 
operation  g^  ^  in  W ^  and  some  nonnegative  function  h(y,v,k,g^  ^) ,  such  that: 

0  £  h(y,v,k,g£  )  £  e(y,v,k,g£>  ;  ¥  (26) 

We  will  show  that  h(y,V,k,g»  )  is  concave  in  y;  thus  if  M  is  some  convex  and 

*-■»  v 

closed  family  of  measures  y,  there  exists  some  y  in  M,  such  that: 

h(y,v,k,g£  )  £  h(y*,v,k,g£  *)  ;  V  ye  III  (27) 

If  the  pair  (y  ,g»  *)  that  satisfies  (27)  happens  to  be  such  that: 


X  ^ 

My  >v»Mg£>v*)  =  My  ,v,k,g£^*) 


(28) 


I 


s 


then  it  is  clear  from  theorem  1,  expression  (27),  lemma  1,  and  expression  (16), 

that  the  pair  (y*, g„  *)  is  also  the  solution  of  the  game  on  F(k,g»  *)xM  with 

X-  y  £,y 

payoff  function  e(y,V,k,g„  *,{a.}).  That  is,  the  pair  (y*,g«  *)  satisfies  then 

i  "V )  y 

the  following  saddle  game  expression: 


VyeM  ;  e(y,v,k,g^  y*,A**  )  <  e(y*,v,k,g^  ^*)  <  e(y*,v,k,g^,{ai»  ; 


y  ’V’s£,y* 


;  where  A  * 

U  ,V,g£  i 


;  V(g£, {a^})  £  F(k,£) 
is  given  by  (15) . 


(29) 


However,  the  equality  in  (28)  may  not  be  satisfied  in  general.  Then,  the 

quantity  h(y*,V,V,gp  *)  in  (27)  will  provide  just  a  lower  bound  on  the  saddle 

-t,y 

value  e(y  ,v,k,g,  *)  of  the  game  in  (29). 

*t,  y 

We  will  now  proceed  with  the  search  for  the  operation  g^  ^  in  and  the  study 
of  the  quantity  h(y,v,k,g„  ).  We  will  first  consider  finite  sequences  Z^  of 

't,  y  i 

observation  data,  and  then  asymptotically  long  such  sequences. 


As  in  section  3,  let  the  smoother  or  filter  operate  on  the  limited  length 


sequence  Z of  observation  data.  Then,  the  form  of  the  smoother  or  filter 


is  as  in  (11).  Let  8^  (z’S  and  6  .  (z'S  be  as  in  (3)  and  (4)  respectively 

X  y  y  y  V  |ijVy  \,K./  ylyj 


Let  us  define: 


Mi-j (y,v)  "  Mij(;j’v)  Ep>v  )8y,v,(k),-n,m-n-l(Zik+l-£)  8y,v, (k) ,-n,m-n-l(Zjk+l-^ j 


(30) 


;  where  M^(y,v)  =  (y,v)  due  to  the  stationai  ity  assumed,  *  means  conjugate,  and 

T  means  transpose. 
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Clearly,  the  matrices  M^_j(y,v)  In  (30)  are  square,  and  nonnegative  definite. 
Consider  the  matrix  Mg(y,v)  and  define: 

AmqCp.v)  :  it  minimizes  with  respect  to  the  vector  A  the  error  expression 

EU,V  jX0  -  A  8v,v,(k),-n,m-n-l(Z^j  (31) 

h(y,v,k,g£^)  =  XZ  -  ^(y,v)M0(y,v)AM  (y,v)  (32) 

Lemma  5 

Let  y  be  given,  and  let  e(y,v,k,g^)  be  the  infimum  in  (16). 


*e,p  :  »*,„<**>  '  8p,v,(k),-„,D-„-i(^) 


Then, 


0  <  h(y,V,k,g^  )  <  e(y,v,k,g£)  ;  V  g^  e  Nl 
;  where  h(y,v,k,g^  y)  is  given  by  (32) 


The  proof  of  the  lemma  is  in  appendix  C.  The  result  in  it  corresponds  to 

O 

expression  (26)  in  the  gist  of  the  approach.  We  observe  that  g„  (z )  is  a  linear 

•c,y 

combination  of  conditional  expectations  of  the  type  given  by  (3)  .  We  now  proceed 
with  the  next  step  in  our  approach,  by  stating  another  lemma  whose  proof  can  be 
found  in  appendix  C. 

Lemma  6 

The  error  h(y,v,k,gp  )  in  (32)  is  strictly  concave  in  y,  on  some  convex  and 
closed  family  M  of  measures  y.  Therefore,  there  exists  a  unique  y*  in  M  such  that 

Mu  ,v,k,g^ M*)  >  h(y,v,k,g^,  ^)  ;  v  yeM 
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We  now  summarize  parts  of  our  discussion  in  this  section  in  conjuction  with 
some  straightforward  observations,  in  a  corollary. 

Corollary  2 

Given  a  convex  and  closed  class  M  of  measures  y,  given  the  sequence  of 
observation  data  on  which  the  filter  or  smoother  operates,  the  supremum  h(y*,V,k,g 
in  lemma  6  bounds  from  below  the  saddle  value  e(y  ,V,k,g^)  of  the  restricted  game 
in  theorem  1,  for  fixed  k,t,  and  all  choices.  Furthermore,  the  lower  bound 

represented  by  h(y  ,v,k,go  *)  is  in  general  tighter  than  the  lower  bound  given  by 
the  saddle  value  e(y  ,v,a  *  ..)  of  the  unrestricted  game  in  theorem  3  and 

M  9  ^  9  J  ) 

corollary  1,  for  filters  or  smoothers  in  the  class  F(k ,£)  .  The  two  bounds  approach 
each  other  as  t  increases,  becoming  asymptotically  identical. 


Let  us  now  consider  the  asymptotic  case,  where  the  observation  sequences  used 
in  the  smoothing  or  filtering  extend  to  infinite  past  and  infinite  future.  Then, 
we  will  use  the  notation  and  the  procedure  used  in  section  3.  Let  us  define  the 
following  Fourier  transforms: 


CO  0O 


.  Y,  Y  ^n>'~3mx  e  U>  (zL>  8*(k)  (z£)| 

^  y,v  (  n,y,v  m,y,vv  J\ 


(33) 


m=-°°  n=-°° 


-  Y 


e-JnX  E 


n=-«> 


j 


6(k)  (zl) 

n,y,vv 


(34) 


Then,  the  asuraptotic  coefficients  {a.}  of  the  vector  A„  (y,v)  are  given  by 

1  "o 

the  following  expression: 


A-  / 

•MT 


H(w)  Sy  v  g(w,X)  dw 


(35) 
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The  error  h(y,v,k,g»  )  in  lemma  6  takes  then  the  following  form: 

V 


IT 

h(y,V,k»g£>y)  =  Ey  X0  “  2tt  f 

•L-n 


IFF  /  H  (w)  Sy,v,B(w)  dw 


;  where  H(w)  the  Fourier  transform  of  the  filter  (an)  given  by  (35)  . 

As  in  section  3,  the  properties  of  h(y,V,k,gn  ),  expressed  by  lemmas  5  and  6, 

are  also  valid  in  the  asymptotic  case.  Thus,  if  M  is  a  concave  and  closed  class  of 

measures  y,  the  sup  h(y,v,k,gp  )  exists,  it  is  unique,  it  is  realized  by  some  y 
yeM 

A  ~ 

in  M,  and  y  is  such  that: 

h(y*,V,k,g»  *)  =  sup  h(y,v,k,g»  ) 

yeM  ^',1- 

Also,  the  conclusions  in  corollary  2  are  still  valid  in  the  asymptotic  case. 

We  will  conclude  this  section  with  the  consideration  of  the  special  classes 
of  information  processes  that  were  studied  in  lemmas  3  and  4,  in  section  4.  We 
will  express  our  conclusions  in  a  lemma  whose  proof  is  in  appendix  C. 

Lemma  7 

Let  the  noise  channel  [R,V,R]  be  either  deterministic,  or  Gaussian,  stationary, 
additive,  and  with  measure  independent  of  the  measure  of  the  information  process. 

Let  M  be  either  one  of  the  following  two  convex  and  closed  families  of  stationary 
processes : 

1.  is  the  class  of  processes  with  identical  means  and  spectral  densities. 

2.  M2  is  the  class  of  processes  with  identical  means  and  with  spectral 
densities  lying  within  a  convex  and  closed  family  S  of  spectra. 

For  both  the  above  families  M,  the  measure  y  in  lemma  6  is  the  Gaussian 
measure,  with  spectral  density  uniquely  determined  for  the  class  in  2. 
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Comparing  the  result  in  lemma  7  with  the  results  in  lemmas  3  and  4,  we  see 
that  for  both  the  considered  families  M,  the  two  lower  bounds  represented  by 
theorem  3  and  lemma  6  are  evaluated  at  the  Gaussian  measure.  Furthermore,  the 
first  bound  is  exactly  equal  to  the  mean  square  error  induced  by  the  Gaussian 
measure  on  F.  Then,  the  properties  of  qualitative  robustness  [10],  [11],  [12]  are 
in  general  violated  if  the  filters  or  smoothers  that  correspond  to  the  above  lower 
bounds  are  selected.  This  is  so,  because  boundness  and  asymptotic  continuity  (as 
in  [10])  are  not  in  general  satisfied  then. 


6.  The  Asymptotic  Lower  Bounds  for  A  Class  M 

In  this  section  we  will  evaluate  the  lower  bounds  of  theorem  3  and  lemma  6, 
for  the  first  class  of  information,  process  in  lemma  7,  and  for  the  same  noise 

channel  specifications  as  in  the  lemma.  We  will  consider  the  asymptotic  case,  and 
for  the  bound  in  lemma  6  we  will  assume  arbitrary  sliding  step  k,  and  finite 
arbitrary  length  £  of  the  sliding  block  window.  We  will  assume  zero  mean  informa¬ 
tion  and  noise  processes. 

Let  us  denote  by  f^(w)  and  Cw)  the  spectral  densities  of  the  noise  and  each 
information  process  y  in  M^,  respectively.  Let  us  also  denote: 


v(m) 


V(m) 


^  I 


O  y  mK-  ( 

“1 -£  mk+l-£J 


E  f Y°  Y  ^ 

V  j  l-£  mk+1 -£ j 


(37) 

(38) 


The  expression  in  (37)  is  not  y-dependent  within  the  class  M^.  Considering 
the  asymptotic  case  with  infinite  past  and  future,  and  due  to  lemma  3,  we  have  a 
well  known  expression  for  the  lower  bound  in  theorem  3.  Indeed,  if  y  is  the 
Gaussian  measure  within  the  class  ,  this  lower  bound  is  given  by  the  following 
expression : 
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e(u*>v,g£)  =  sup  e(y,v,a  «,)  =  4r  /  f  (w)  dw 
yeM  y’v’  zir  /  y 


f2(w) 

_y__ 

(w)+fv(w) 


dw  (39) 


We  now  proceed  with  the  evaluation  of  the  lower  bound  in  lemma  7,  for  the  class 
From  the  proof  of  lemma  7,  in  appendix  C,  we  have: 

■  em  jxo  !V<°>  +  V<0)rl  ^  •  v  “eMx  <4« 

(k)  l 

;  where  in  y  ,,(z  )  is  not  y-dependent. 
l  m,y,v 

*  (k)  t 

Also,  for  the  Gaussian  measure  y  ,  we  have  for  6  (z  )  defined  by  (3): 

m,y,v 

fi(k)  (JU  A  (k)  ( JL  „  .... 

Sm,|i*,v<z  >  -  Yn,y,v<z  >  :  V  "  (41) 


Using  expressions  (40)  and  (41) ,  we  have  then: 


Sy(k)  (z£).v*(k)  e  !e(k)  (7l)  q*(k>  (7li\ 

'Yn,y,v(Z  )\y,v(Z  )(-  V.v  8n,y*,v(Z  >  3m,y*,v(Z  >  = 


Ey,v  j^y.v^-Vy.v 

-  I?  )v  l 


(  T  |  (  T  ) 

M  |X0  Xnk+l-£ |  'V(0>  +  V'0”'1  Ep  jX0  Xm“il-£| 


Let  (ri(y)}  be  the  autocovariance  coefficients  determined  by  the  spectral 
density  f^(w),  within  the  class  Those  coefficients  are  y- independent  within 

the  class  .  We  then  obtain: 


00  / 

l  T 

V  e--^ nx  E  X  X  nk 

C  y  (X0  \k+l-Z 


10°  00 
■  E  e’J,“  r„k(y) . E  e'jnlt  r„k-£+i(i,)  <43) 

—  n  = OO  r\  — ** 


But  after  some  manipulations,  we  obtain: 
00  k-1  x+27ri 

E  .-1”  WU,  E  .'J  ^  ’ 


;  0  <  m  <  l-l 


n=-oo 


e 


x-t-27ii 

k 
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Substituting  expression  (44)  in  expression  (43),  we  also  obtain: 


V  e-j™  E  xn  X 
'  U  1  0 


X  xnk  1 

y  X0  nk+l-£ 


1  \  / x+2tt  i  \  r  ,  ~ 

■iEM~r/[1 . e 


.  X+27Ti 


-3  ^  a- 1) 
,e  *  J 


We  now  substitute  expressions  (45)  and  (42)  in  expression  (33),  to  obtain  in  a 


straightforward  manner: 


i=0  n=0 

.j  _  m  -j  (£-1)  _ 

•  [1,  •  • .  >e  K  , . . . , e  K  ]  [Ryp(0)  +  3^(0)]  • 

.  A+2-rrn  .  A+2irn  ,  »  ,  N 

j  - r-  m  j  — j—  (£-1)  T 

• [1 , . . . , e  K  , • . . , e  K  ]  (46) 

For  arbitrary  finite  t  and  some  k  such  that  1  k  <  £,  the  lower  bound  in  lemma 
6  will  be  given  in  this  case  by  the  expression: 


h  C n  y  ^  ) 


i  y 

2tt  /  fy 

•'-7 r 


(w)dw  -  (2ir) 


!/hw 


)  S  *  r(A,x)  H  (A)  dx  dA  (47) 

y  ,v,b 


;  where  S  *  R(A,x)  is  given  by  (46),  and  H(x)  is  the  solution  of  (35)  with 
M  y  v  P 

S  R(X)  determined  by  (z'S  =  y^.  ..(z'S  . 

y,  ->,p  m,y,v  m,y,v 

It  is  easy  to  see  that  for  k  =  t  =  1  the  bound  h(y  ,V,l,g  *)  is  the  minimum 

oo  1  *1* 

error  E  j Xn  -  Y]  a.  Z.  J  ^  with  respect  to  the  filter  {a,}.  Therefore, 

y»v  /  u  i=_m  1  1  I  1 

h(y  ,v,l,g  .)  is  then  identical  to  the  global  lower  bound  e(y  ,v,g  )  in  (59). 

9  M  *  00 

This  was  expected  since  both  bounds  are  realized  at  the  Gaussian  measure  y*,  and 
for  this  measure  the  error  minimization  on  F(l,l)  is  identical  to  the  error 


minimization  on  F  and  F(L) . 
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£ 

As  in  lemma  7,  the  bound  h(y  ,v,k,g^  ^*)  in  (47)  is  a  strictly  concave  function 
of  f  (w)  ,  for  all  k  and  £.  Therefore,  if  the  class  in  lemma  7  is  considered 
rather  than  the  class  M^,  there  will  be  a  unique  spectral  density  f^,  which  will 
maximize  the  bound  h(|i  ,v,k,g„  *)  within  the  convex  and  closed  family  5  of 

t  M 

spectral  densities.  This  maximum  value  will  be  the  lower  bound  of  lemma  7  in 

ic  k 

then.  Similar  argument  holds  for  the  global  lower  bound  e(y  ,v,g  )  of  (39),  in  . 

OO  / 

7 .  Conclusions 

Given  some  convex  and  closed  family  M  of  information  processes,  we  developed 
two  classes  of  lower  bounds  on  the  performance  of  filters  and  smoothers  in  M.  The 
mean  square  performance  criterion  was  adopted.  The  developed  bounds  are  functions 
of  conditional  expectations  of  the  type  expressed  by  (3) .  Thus,  they  are,  in 
general,  functions  of  high  order  statistics  in  /•'.  In  other  words,  these  bounds  are, 
in  general,  tightly  associated  with  the  family  M,  in  measure  rather  than  in  spectral 
characteristics  only.  So,  if  the  class  M  is,  for  example,  described  by  the  linear 
contamination  of  a  nominal  measure,  and  if  the  Gaussian  measure  is  not  included  in 
M,  then  the  developed  lower  bounds  will  be  nonlinear.  Therefore,  the  adoption  of 
linear  filtering  (as  in  [13])  will,  in  general,  result  in  inferior  performance 
across  M  then. 
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Appendix  A 


Proof  of  Lemma  1 


2 

In  the  expression  of  e(y,V,k,g£)  in  (14),  the  term  v  XQ  is  linear  with 


respect  to  y.  Thus,  we  must  only  show  that  the  expression 

£  Ep>v  |x0  V^(Z)  |  Eu_v  jx0  ,g«j 

is  convex  in  M. 


(A.l) 


Let  y^£  M  and  y2  £  M.  Then,  since  M  is  convex  and  closed,  for  any  e  :  0  <  £  <  1 
we  also  have  [£y^  +  (l-e)y2]  £  M. 

To  simplify  the  notation,  let  us  denote: 


!i  ■  v  |xo  y  z>j  >  *  -  i.* 


R±  =  iKy^v.g^)  ;  i  =  1,2 


(A. 2) 


Then,  it  is  clear  that: 


R(eyi+(l-e)y2,v,g^>  =  e  r  +  (1-e)  r2 


(A. 3) 


Substituting  (A. 3)  in  (A.l),  we  find: 

K(£y1  +  (l-£)  y2,V,g^)  =  [£  Ex  +  (l-£)  E21T  [e  R1  +  (l-£)  R2]_1  [e  E1  +  (1-e)  E2] 


(A. 4) 


For  convexity,  we  want  to  prove  that: 


~2  K<e  Pi  +  0-0  Mo.v.g-) 

:  ^  £  =  0 


(A. 5) 


I 
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We  will  need  the  following  known  equation: 


M_1(x)  =  -  M-1 (x)  r|^  M(x) ]  M_1(x) 


(A.  6) 


;  where  M(x)  a  nonsingular  square  matrix,  and  x  a  scalar  variable  appearing  in  the 
components  of  the  matrix  M(x) . 

Differentiating  the  expression  (A. 4)  twice  with  respect  to  £,  and  through  the 
application  of  expression  (A. 6),  we  find  after  some  manipulations: 

2 

^~2  K(e  +  (1-e)  y2,v,g£)  =  2[E1  -  r"1  e2]T  R^1  [E1  -  ^  r"1  E2]  (A. 7) 

The  expression  (A. 7)  is  clearly  nonnegative  since  R  ^  is  a  nonnegative  definite 
matrix  (correlation  matrix).  Thus,  (A. 5)  is  satisfied  and  the  proof  is  complete. 

Proof  of  Theorem  1 

Due  to  lemma  1,  there  exists  some  y*  in  M  that  satisfies  the  supremum 

sup  e(y,v,k,g„).  Even  if  y*  is  not  unique,  for  every  £  :  0  <  e  <  1  and  every 

yeM  we  have  that  [(1-e)  y*  +  e  y]  e  M.  This  is  due  to  the  convexity  and  closeness 

of  the  set  M.  Also,  even  when  there  are  more  than  one  measures  in  M  that  satisfy 

the  supremum  sup  e(y,v,k,g») ,  there  exists  among  them  at  least  one  y  ,  such  that: 
yeM 


e((l-e)y*  +  e  y,v,k,g^)  <  0  ;  V  yeM 


(A.  8) 


Due  to  (16),  and  using  the  expression  in  (14),  we  easily  obtain: 


9_  e(l-e)y*  +  e  y,v,k,g„)  =  -  e(y  ,v,k,g„)  +  e(y,v,k,g/>,{at» 

Se  e  =  0  1 


(A. 9) 


;  where  {a*}  =  A**  ,  and  A"*  given  by  (15). 

±  M  9  V  9  M  >  V  9 

From  (A. 8)  and  (A. 9)  we  conclude  then: 

There  exists  some  y*  in  M  that  satisfies  the  supremum  in  lemma  1  and 


is  such  that: 


e(u,v,k,g£,{a*})  <  e(y*,v,k,g£,{a*})  ;  V  yeM 

The  part 

e(y*,v,k,g£,{a*})  <  e(y*,v,k,g^, {ai»  ;  V  {aj  e  FCk.g^) 

£ 

is  trivially  true  since  {a^}  satisfies  the  infimum  in  (16)  at  the  process  y* 
The  proof  of  the  theorem  is  now  complete. 
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Appendix  E 


Proof  of  Lemma  2 


Let  belong  to  M.  Let  e  :  0  <  e  <  1.  Then,  since  M  is  convex  and 

closed  we  also  have  [e  y^  +  (l-e)y2]  e  M.  Using  the  symbol  f  ^  for  t*ie 

(j-i) -dimensional  density  function  induced  by  y  and  V,  as  introduced  in  section  2, 


we  easily  iind: 


«  Ef»1.v(zi'i)a|.1,.,(U)<lJ‘1)tU-£,f 

%  +(l-e)n  ,v,(i,j)u  ; - ± - ±7- - — 


(B.l) 


Directly  from  (B.l)  we  also  find: 


£  %1+(l-£)y2,v,(i,j)u  >  =  — 


yrv,(i,j)(zj  i}  ‘  ay2,v,(i,j) 


|efiy',(*J’i)+<1-e>tVv(l3'I>1' 


(B.2) 


2  a£u1+(l-e)y2,v,(i,j)<z'1  2[fP1,v(z'i  *  cu2,v(zj  )]£m1,v<zJ  )fu2,v<zJ  *[“p  v,(i,j) 


%„v,(i,i)(zj'1,][efp,,v<zJ'i>+(1-s>fp„v<z;l'1)l 


1_  Q.  (Zi-i) 

h  de  ejJ1+(l-£)]i2,v,  (i,  j)  v 


b 


a 


3e2  £M1+(l-c)U2,v, (i,j) 
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fv,1,v('3'i>  r  /  ^ 

e  =  °  fy  v(zj~1)  t- y1»v,(i,j)( 

—  O  x 

n  *  __ 

e  -  0  fy,.vf2j'I> 

(B.4) 


ay1,v,(i,j)(z3  1}  ■  ay2,v,(i,j)(z 


j_i)j  (B 


.5) 


1 


3£2  ey1+(l-£)y2,v  j  ey^U-e^.v,  (i,  j) 


(zj-1)  (-  4  E. 


yl-y2,V  j  eH1+(1-£:>^2,V,(i»:5) 

1-  a  (zj"1)) 

3e  aey1+(l-£)y2,v,Ci,j)^  >( 


(z^) 


+  2  Eey1+(l-e)y2,vj^3e  aey1+(l-E)y2,v> (i,j) (z  * 


+  aey1+(l-e)y2,v,(i,j) ^Z'  ^  *  3£2  aey1+Cl-e)y2,v, (i, j) ^  (B.6) 


Applying  expressions  (B.2)  -  (B.5)  in  expression  (B.6),  we  finally  find: 


2  "r.y1+(l-e)y2,v  jct£;y1+(l-e)y2,v,  (i, j ) 


3e 


(z-1-1); 


£  =  0 


=  2  E  \ 
yl 


V'^ 


i;  ‘  “i<2,v,(i,j)(l!3  l) 


(B .  7) 
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The  expression  in  (B.7)  is  nonnegative.  It  is  strictly  positive  iff 

a  ,  .  (z^-:S  and  a  ,,  ..(z^  are  not  equal  almost  everywhere  in 

Thus,  E  i  ..(zj-i)>  is  convex  in  M.  Since  E  is  linear  in  M,  we 

y,v  \  y,v,(i,j)  1  y  0 

conclude  that  e(y,v,  a  ,  .  )  is  concave  in  M. 

y.v, » J ./ 

Proof  of  Theorem  3 

According  to  lemma  2,  there  exists  some  y  in  M  that  realizes  the  supremum  of 
e(y,v,  ^  in  M.  Even  if  there  are  more  than  one  such  measures,  there 

exists  at  least  one  such  that: 

For  every  £  :  0  <  £  <  1,  and  every  y  in  M: 


M,v,  a(1_e)llV_  u,v,(i,J))|e  ,  „  ±  0  ; 


(B  .8) 


Notice  that  since  M  is  convex  and  closed,  [(l-£)y  +£y]  £  M.  But  from  (24)  we  have: 
e  |(l-e)y  +e  y,v,  ct(1_£)^*+£  y,v,,(i,j))  E(l-£)y*+£  y  X0  E(l-e)y*+e  y,v  |a(l-e)y*+e  y,v,(i,j)^ 


(B.9) 


Applying  then  the  results  in  (B.1)-(B.5)  in  (B.9),  we  find: 


£  =  0 


-r—  e  f(l-£)y*+e  y,v,  a,,  .  *  , 

3c  '  ’  ’  (i-£)y  +£  y,v,(i,j); 


\ 

=  “[v  x0  “  Ey*,v  '  ay*,v,(i,j)fzj  }(]+  Ey  X0 


EP,V  - 


1  ,  .  f  „(z* 

2  E  *  |  a*  „  „  ^(Zj_1)  - 


i)  [a 

l  y, 


V.v  )  y  ,v,(i,j) 


(g  3  1) 
_ 1 

f  *  (r»^ 

y  ,vv"  } 


V.Ml, 


.yzi-1)] 


(B.10) 


I 


*„  V.v.a.o6  vt 

'  fy*,vc  ' 

1  V.».(ij)t,1'i)  ^V.v.d.D 
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•c^v-  V.v.(ij)>-Vv  k-V.v.aj)^) 


Xo  "  2  Ey,v  ay,v,(ij)(z 
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CB.13: 


Substituting  (B.ll),  (B.12),  and  (B.13),  in  (B.10),  we  obtain: 


9  /  * 

e  [  (l-e)y  +e  y,v,  a. 


(l-e)y*+e  y,v,(i,j))  £  =  Q  “  e(y’V’  ay*,v,(i,j))  ~ 


-  e(l1  -v-  V,v,(i,j)> 


(B.u; 


Finally,  from  (b.8)  and  (B.14)  we  obtain: 


e(u>v>  V,v,U,j))  s  e(lI*>v'  V,V.(1.J))  !  v  ^ 


(B.15) 


Expression  (B.15)  proves  the  left  part  of  the  game.  The  right  part  is 
trivially  satisfied  due  to  (24) . 

Proof  of  Lemma  3 

Consider  the  linear  class  F(L)  of  filters  and  smoothers,  as  introduced  in 

section  2.  Let  us  denote  by  e^y.v)  the  infimum  of  the  error  in  (6)  in  F(L),  at 

the  neasure  y.  Then,  due  to  the  fact  that  e  (y,V)  is  only  a  function  of  first  and 

L 

second  order  statistics,  we  have  that  for  any  y^,  y2  in  the  class  M  of  the  lemma: 

eL(y1,v)  =  eL(y2,V)  =  eL(y*,v)  ;  V  y1,y2  e  M  (B.16) 

^Hf 

;  where  by  y  we  have  denoted  the  Gaussian  measure  in  M. 

Bue  due  to  (9)  we  have: 


eL(y,v)  >  e(y,v,  a  )  ;  V  yeM 
Also,  for  the  Gaussian  measure  y*  we  have: 


(B.17) 


eL(y*’V)  =  e(y*>V’  (B.18) 

A 

;  where  (B.18)  is  true  since  y  and  V  jointly  Gaussian  here. 

Thus,  from  (B.16),  (B.17)  and  (B.18),  we  obtain: 

e  (y*,v)  =  e(y*,v,  a  *  )  >  e(y,v,  a  )  ;  V  yeM 

L,  M  » v  v 


Thus,  it  is  clear  that  the  pair  (y  ,  ^)  is  a  solution  of  the  game  in  theorem 
3.  The  solution  is  unique  if  for  any  y^,y2  ^  the  conditional  expectations  ^(z) 
and  a  (z)  can  not  be  equal  almost  everywhere. 

u2 
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>endix  C 


Proof  of  Lemma  5 


For  any  g^,  the  infimum  e(y,v,k,g^)  is  given  by  expression  (14).  We  see  in  a 


relatively  straightforward  manner,  that  the  following  equation  holds  ;  where  A* 
is  given  by  (15) : 


P,V,g£ 


S..V  |X0  Hm>v  |x0  y»)  -  jg^)  ^v>g£ 


(C.l) 


Let  A  be  an  arbitrary  vector.  Then,  substituting  A  by  A  in  (C.l)  and 

V* » v , 

applying  the  Schwartz  inequality,  we  obtain: 


Vv  |^V  8p ,  V ,  «  k)  ,  -n ,  m-n-l (Z  £)J  i  Ep ,  v  8p,V,  00  | } 


.  rl/2  j  |  JLi  2  I  1/2  (, 


k  b  a1)  2 

y ,v, (k) -n,m-n-l  I 


1/2  !l 

=  Eh.v  j 8£(Z 


[Ax  M0(y,v)M 


(C.2) 


with  equality  everywhere  in  (C.2)  iff: 


gAz1)  ^CAT8  ,  »  .(zS 

y,V, (k) ,-n,m~n-l' 


(C.3) 


For  g^(z  )  as  in  (C.3),  the  upper  bound  in  (C.2)  is  maximized  for  vector  A 


such  that  it  minimizes  the  mean  square  error  expression 

(  t  l  )  2 

E,  ,,  <X  -  A  B  ,  .  ,(Z  )  .  The  proof  is  now  complete. 

y,v  |  0  P,v, (k) ,-n,m-n-l  J 

Proof  of  Lemma  6 


We  will  present  the  proof  in  two  steps. 


i)  Consider  the  matrix  Mq(u,v)  in  (30)  .  Let  A  be  some  arbitrary  m- 

dimensional  constant  vector.  We  will  show  that  the  quadratic  form 
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A  M  (y,v)A  is  convex  in  y  on  M.  We  will  refer  to  the  proof  of  lemma  2, 


in  appendix  B.  Let  y^,y2  G  Xet  e  :  ®  *  £  <  Then,  [(l-£)y2  +  E  y^Jc  M.| 

(k)  / 

It  can  be  easily  seen  that  each  g'  '  .  .  (z  ;  in  the  matrix 

i * M2» v 

MQ(e  y^+(l-e)y2,v)  has  fche  same  form  as  in  (B.l).  Then,  using  the 


expressions  (B.1)-(B.5)  and  differentiating  the  quadratic  form 


A  Mq(e  y1+(l-e)y2,v)A  twice  with  respect  to  £,  we  obtain: 


3e 


2  A  Mq(£  y1+(l-£)y2,v)  A 


=  2E  V  a.  [gfk)  (Z^)  -  g.  (Z*)l 
=  o  1  i,y2>vv  ;J 


(C.4) 


;  where  {a.}  the  components  of  the  vector  A. 


The  expression  in  (C.4)  is  strictly  positive  for  nonzero  A  and  notrivial 


measures.  Thus,  A  M  (y,v)A  is  then  strictly  convex  in  y  on  M. 


ii)  We  will  show  that  A^  (y,V)MQ(y , VjA^Cy.v)  is  strictly  convex  in  y  on  M. 

2 

Since  E  X  is  linear  in  y,  this  will  also  show  that  h(y,v,k,g„  )  is 
y  C  Ij  y 

strictly  concave  in  y  on  M.  Let  y^,y2  £  M*  Let  E  :  0  <  e  <  1,  Then 


[(l-e)y2+£  y^]  e  M.  Due  to  part  i)  of  the  proof,  we  have: 


A  .  T 


XM  y1+(l-£)y2>v)  =  A,V|  y1+(l-e)y2,v)M0(£  y1+(l-£)y2,v)AM  (e  y1+(l-e)y2,v) 


<  £  Aw  (£  y1+(l-e)y2,v)M0(y1,v)AJlj  (£  y1+(l-e)y2,v)  + 


+  (1-e) A,.  (£  y  +(l-E)y9,v)Mn(y9,v)AM  (e  y1+(l-£)y9,v) 
M0 


2 ,  ; '  o v M2 *  ; amq w  -r 


(C  .5) 


But,  by  definition: 


aM  y1+(l-E)y2,v)M0(yi,v)AH  (£  y1+(l-£)y2,v)  <  A^  (y1,v) 


aMq(e  y1+(l-E)y2,v)M0(y2,v)AA)  (e  y1+(l-c)y2,v)  <  A^  (y2,v) 


(C.6) 
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From  (C.5)  and  (C.6)  we  thus  obtain: 


XMq(E  y1+(l-e)y2,v)  <  e  X^ty^v)  +  (1-e)  X^(y2,v) 


(C.7) 


;  where  X^  (y,v)  =  (y,v)MQ(y,v)  (y,v) 


Proof  of  Lemma  7 


i)  Given  y,  let  us  define: 


’  Eu  !  x0  xI“+l-i  1 V  +  "vi1'1 


R  =  E  )  x"*  X 

y£  y  )  mk+1— i  mk+1— 2 


R\>1  EV  )Ymk+l-£  Ymk+l-£ 


(C.8) 


(C.9) 


(C.10) 


The  (z^)  is  the  linear  mean  square  estimate  of  X0,  given 

m,y,v  0  mk+1— c 

For  the  Gaussian  measure  y*,  (z'S  is  exactly  the  same  with  the 

B^y  v(z^)  in  (3)  •  If  in  the  matrix  M^Cy.v)  we  substitute  3^^  v(z^) 

by  yK  ,.(z  ),  we  will  obtain  a  matrix  M_  (y,v)  with  the  same  convexity 
in,  m  > v  u  ,y 

properties  as  in  M  (y,v)  and  with  a  maximum  inner  product  X..  (y  v) 

0  m0,y 

(denoted  in  the  proof  of  lemma  6) ,  that  is  in  general  lower  than  the 

inner  product  X.,  (y,\>)  in  (32)  .  This  is  clear  from  the  proofs  of  lemmas 
0 

5  and  6.  Also,  the  components  of  the  matrix  Mn  (y,V)  are  strictly  functions 

u  ,y 

of  covariance  coefficients  (see  (C.8)),  thus  the  matrices  Mn  (y,v)  are 

u  ,y 

identical  within  the  class  Thus,  also: 


\,Y<VV)  ■  XM0,Y(U2'V)  1  V  e  *'l 


(C.ll) 


Also,  given  y  in  : 


Y^>v)  <  Hi  (y’v) 

VY  m0 


(C.12) 


I 
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If  y*  the  Gaussian  measure  in  we  finally  obtain  combining  (C.ll) 
and  (C.12) . 


XMQ(y*’V)  =  XM0,Y(y*,V)  -  XMtfY  (y*v)  ;  V  yeMl 


(C.13) 


This  proves  the  lemma  for  the  class  M^. 

ii)  As  in  part  i)  of  the  proof  of  lemma  6,  for  any  vector  A  the  quadratic 
T 

form  A  M  (y,V)A  is  strictly  convex  in  y.  It  is  also  a  function  of 
U  »Y 

spectral  characteristics  only.  Thus,  it  is  strictly  convex  in  f  on  S; 

where  f  the  spectral  density  of  the  process  y.  Following  exactly  the 

same  procedure  as  in  part  ii)  of  the  proof  of  lemma  6  we  can  then  show 

* 

that  there  exists  a  unique  f  in  S  such  that 


±  <W>V)  ;  V  £  CS 


(C.14) 


;  where  A  .  (y,v)  the  inner  product  A.,  ..  (y,v)  if  the  process  y  has 

Mq.Y.i  Mq»  r 

spectral  density  f. 

Thus  the  measure  y  for  the  class  °f  the  lemma  is  the  Gaussian  with 
spectral  density  f*,  signified  by  (C.14). 


The  dimensionality  of  the  observation  sequence  was  never  used  in  our 
proof.  Thus,  our  results  hold  asymptotically,  as  well  as  nonasymptotically . 
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